Multi-focal structured illumination microscopy systems and methods

ABSTRACT

Various embodiments (300, 400, 500) for a multi-focal selective illumination microscopy (SIM) system for generating multi-focal patterns of a sample are disclosed. The embodiments (300, 400, 500) of the multi-focal SIM system perform a focusing, scaling and summing operation on each generated multi-focal pattern in a sequence of multi-focal patterns that completely scan the sample to produce a high resolution composite image.

FIELD

This document relates to multi-focal structured illumination microscopy,and in particular, to multi-focal structured illumination microscopysystems and methods for producing a plurality of multi-focal fluorescentemissions resulting from multi-focal patterns of a sample.

BACKGROUND

Classical fluorescence microscopy is limited in resolution by thewavelength of light, referred to as the “diffraction limit”, whichrestricts lateral resolution to about 200 nm and axial resolution toabout 500 nm at typical excitation and emission wavelengths when asample emits fluorescence that is detected by the microscope. Confocalmicroscopy is an optical imaging technique used to increase opticalresolution beyond the diffraction limit by using point illumination anda spatial pinhole arrangement to eliminate out-of-focus emission lightfrom specimens that are thicker than that of the focal plane, therebydelivering images with 1.41 times the resolution than the diffractionlimit by a method that requires tightly closing the pinhole.Unfortunately, closing the pinhole diminishes the signal level of theemitted light from the sample to such an extent as to make thisparticular method of super-resolution impractical. In addition, aconfocal microscope must perfectly align the excitation from themicroscope's illumination beam with the pinhole/detector, since amisaligned pinhole results in a reduced and weak light signal beingdetected as well as resulting in reduced axial optical sectioning of thesample itself. As such, misalignment of the confocal microscope cancause a reduction in the light signal.

A method for resolution enhancement for confocal microscopy has beenfound that uses an array of detectors, such as pixels in a camera image,wherein each of the detectors in the array produces a separate confocalimage. If the array of detectors is sufficiently small, each of theformed confocal images can be equivalent to similar confocal imagesformed by a confocal microscope with a tightly closed pinhole such that1.41 times the resolution of the diffraction-limited microscope isachieved when the confocal images are properly aligned. In addition,deconvolution provides a further increase in image resolution. However,this detector array arrangement is limited since only a singleexcitation point is scanned throughout a two-dimensional plane of thesample, which limits the speed the sample can be scanned and subsequentdetection of the fluorescence emissions of the sample.

Another type of microscopy, referred to as structured illuminationmicroscopy (SIM), illuminates a sample with spatially modulatedexcitation intensity, which is translated and rotated in differentpositions relative to the sample, with a wide-field image being taken ateach translation and rotation. Processing the raw images appropriatelyresults in a final image having double the lateral resolution ofconventional wide-field microscopy. Although such SIM systems generateimages with 2× the spatial resolution of a conventional microscope,there is still a sacrifice in temporal resolution when producing thefinal image, as time is required to acquire each of the multiple rawimages. SIM may also be used to reject out-of-focus blur, known as“optical sectioning”. However, such optical-sectioning is performedcomputationally, and is thus subject to shot (Poisson) noise. SIM isthus inappropriate for thick or highly stained samples, when backgroundfluorescence may cause this shot noise contribution to overwhelm thein-focus signal.

As such, there is a need in the art for a structured illuminationmicroscopy system that produces a multi-focal excitation pattern of thesample for each high resolution image without sacrificing scanningspeed, and that is resistant to the shot noise that may corrupt SIMimages.

SUMMARY

In an embodiment, a microscopy system may include a light source fortransmitting a single light beam and a beam splitter for splitting thesingle light beam into a plurality of light beams forming a multi-focalpattern. A scanner scans the plurality of light beams that forms themulti-focal pattern onto a sample such that the sample generates aplurality of fluorescent emissions resulting from each multi-focalpattern. A focusing component then defines an aperture configured tophysically block out-of-focus fluorescence emissions of the plurality offluorescent emissions resulting from each multi-focal pattern and allowsthrough in-focus fluorescent emissions to a pass through the aperture.In addition, a scaling component scales down the plurality of in-focusfluorescent emissions resulting from each multi-focal pattern such thateach of the plurality of in-focus fluorescent emissions is scaled downby a predetermined factor to produce a plurality of scaled in-focusfluorescent emissions resulting from each multi-focal pattern. A summingcomponent sums each of the plurality of scaled in-focus fluorescentemissions to produce a plurality of summed, scaled in-focus fluorescentemissions that form a composite image of the plurality of summed, scaledin-focus fluorescent emissions.

In another embodiment, a microscopy system may include a light sourcefor transmitting a single light beam; a beam splitter for splitting thesingle light beam into a plurality of light beams forming a plurality ofmulti-focal patterns, wherein each of the plurality of multi-focalpatterns defines a plurality of focal points; a scanner for scanning theplurality of light beams that forms each of the plurality of multi-focalpatterns onto a sample such that the sample generates a plurality offluorescent emissions resulting from each of the multi-focal patterns,wherein each of the plurality multi-focal patterns defines a pluralityof fluorescent focal points; a detector for collecting the plurality offluorescent emissions resulting from each of the multi-focal patterns;and a processing system for processing the collected multi-focalfluorescent emissions from the detector comprising: a processor inoperative communication with a database for storing the plurality ofcollected multi-focal fluorescent emissions, wherein the processorremoves out-of-focus fluorescent emissions resulting from each of theplurality of multi-focal patterns to leave only in-focus fluorescentemissions resulting from each of the plurality of multi-focal patterns,wherein the processor then scales the in-focus fluorescent emissionsresulting from each of the plurality of multi-focal patterns in a localcontraction operation in which each of the plurality of fluorescentemissions resulting from each of the multi-focal patterns maintains thesame proportional distance from another plurality of fluorescentemissions resulting from the multi-focal pattern as the plurality offluorescent emissions contract to produce scaled, in-focus fluorescentemissions; wherein the processor sums the plurality of multi-focalin-focus fluorescent emissions to produce a composite image.

In yet another embodiment, a method for multi-focal structuredillumination microscopy may include:

-   -   generating a single light beam;    -   splitting the single light beam into a plurality of light beams        in which the focal point of each the plurality of light beam        forms a plurality of multi-focal patterns, illuminating a sample        with the plurality of light beams forming each of the plurality        of multi-focal patterns, wherein the illuminated sample produces        a plurality of fluorescent emissions resulting from each the        plurality of multi-focal patterns;    -   performing a pinholing operation in which out-of-focus        fluorescent emissions from the plurality of fluorescent        emissions are blocked and only in-focus fluorescent emissions        from the plurality of fluorescent emissions are permitted to        pass through during the pinholing operation;    -   scaling each focal point of the plurality of in-focus        fluorescent emissions resulting from the plurality of        multi-focal patterns by a predetermined factor to produce a        plurality of scaled, in-focus fluorescent emissions resulting        from the plurality of multi-focal patterns; and    -   summing each of the plurality of scaled, in-focus fluorescent        emissions to form a composite image.

In yet another embodiment, a microscopy system may include a lightsource for transmitting a single light beam. A first microlens arraythat splits the single light beam into a plurality of light beams forforming at least one multi-focal pattern and a scanner that scans theplurality of light beams that forms the at least one multi-focal patternonto a sample such that the sample generates a plurality of fluorescentemissions with each of the at least one multi-focal pattern. Inaddition, a pinhole array blocks out-of-focus fluorescent emissions foreach of the at least one multi-focal pattern and allows through in-focusfluorescent emissions to pass through the pinhole array. A secondmicrolens array produces a non-inverted image of the plurality of lightbeams having a one half a magnification, wherein the scanner rescans thenon-inverted image of the plurality of light beams. Finally, a cameracaptures the scanned non-inverted image.

In a further embodiment, a microscopy system may include a light sourcefor transmitting a single light beam; a spinning disk with a microlensarray for splitting the single light beam into a plurality of lightbeams forming a multi-focal pattern; a spinning disk with a pinholearray for blocking out-of-focus light beams of the plurality of lightbeams for each multi-focal pattern and allowing through in-focus lightbeams of the plurality of light beams to pass through the spinning diskwith a pinhole array, wherein the spinning disk with a microlens arrayis rotated in tandem with the spinning disk with a pinhole array forscanning the plurality of light beams across a sample and generating aplurality of fluorescent emissions; and a camera for capturing an imageof the plurality of fluorescent emissions in each multi-focal patterngenerated by the sample.

In another embodiment, a microscopy system may include a light sourcefor transmitting a single light beam; a first spinning disk withconverging microlens array for rotation in a first direction andpositioned along an optic axis for splitting the single light beam intoa plurality of light beams forming a multi-focal pattern. A secondspinning disk with a pinhole array for rotation in the first directionand positioned along the optic axis for blocking out-of-focus lightbeams of the plurality of light beams for each multi-focal pattern andallowing through in-focus light beams of the plurality of light beams topass through the spinning disk with a pinhole array and a third spinningdisk with a diverging microlens array for rotation in the firstdirection and positioned along the optic axis, wherein the firstspinning disk with converging microlens array, the second spinning diskwith a pinhole array, and the third spinning disk with a divergingmicrolens array rotate in sync relative to each other.

Additional objectives, advantages and novel features will be set forthin the description which follows or will become apparent to thoseskilled in the art upon examination of the drawings and detaileddescription which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram illustrating one method forgenerating a multi-focal pattern in a multi-focal structuredillumination (MSIM) system;

FIG. 2 is a simplified block diagram illustrating another method forgenerating a multi-focal pattern in another embodiment of themulti-focal SIM system;

FIG. 3 is a simplified illustration showing the various components forone embodiment of the multi-focal SIM system of FIG. 1;

FIG. 4 is a simplified illustration showing the various components forone embodiment of the multi-focal SIM system of FIG. 2;

FIG. 5 is a simplified illustration showing a multi-focal pattern with aplurality of focal points; and

FIG. 6 is a simplified illustration showing a scaling operation thatscales in-focus fluorescent emissions resulting from a multi-focalpattern emitted by a sample;

FIG. 7 is a flow chart illustrating one method for conducting a, scalingand summing operation by a processing system in the multi-focal SIMsystem of FIG. 4;

FIG. 8 is a flow chart illustrating one method for extracting latticevectors when executing the out-of-focus fluorescence rejection, scalingand summing operation of the processing system;

FIG. 9 is a flow chart illustrating one method for determining offsetvectors when executing the out-of-focus fluorescence rejection ofout-of-focus blur, scaling and summing operation of the processingsystem;

FIG. 10 is a flow chart illustrating one method for obtaining shiftvectors executing the out-of-focus fluorescence rejection, scaling andsumming operations of the processing system;

FIG. 11 is a flow chart illustrating one method for pinhole masking whenexecuting the out-of-focus fluorescence rejection, scaling and summingoperation of the processing system;

FIG. 12 is a flow chart illustrating one method for scaling and localcontraction when executing the out-of-focus fluorescence rejection,scaling and summing operation of the processing system;

FIG. 13 is a flow chart illustrating one method for summing whenexecuting the out-of-focus fluorescence rejection, scaling and summingoperation of the processing system;

FIG. 14 is a simplified illustration showing the illumination systemused during testing of one embodiment of the multi-focal SIM system;

FIG. 15 illustrates various images of a sample embedded in fluoromount;

FIG. 16 illustrates various images of dual-labeled three-dimensionalsamples embedded in fluoromount; and

FIG. 17 illustrates various images of a sample using the multi-focalillumination system after pinhole focusing, scaling, andthree-dimensional deconvolution;

FIG. 18 is a simplified illustration showing an embodiment for aswept-field hardware MSIM system;

FIG. 19 is an image of a super-resolution images of microtubules withswept-field MSIM system;

FIG. 20 is a simplified illustration showing an embodiment for aspinning disk confocal microscopy system; and

FIG. 21 is a simplified illustration showing an embodiment for aspinning disk MSIM system.

Corresponding reference characters indicate corresponding elements amongthe view of the drawings. The headings used in the figures should not beinterpreted to limit the scope of the claims.

DETAILED DESCRIPTION

In modern microscopy, structured illumination microscopy (SIM) may beused to examine single cells using spatially patterned light to excitesample fluorescence that is later detected and one or more imagesprocessed to produce a super-resolution image with 2× the resolution ofa conventional wide-field microscopy image. However, the SIM systemsacrifices speed for higher resolution (taking multiple raw images foreach super-resolution image). Furthermore, optical-sectioning in a SIMsystem is performed computationally, and is thus prone to shot noiseinherent in fluorescent background. This limits the thickness of thesample that can be examined, thereby requiring other microscopytechniques be used when examining thicker samples. For example, aconfocal microscopy system physically rejects out-of-focus light using apinhole arrangement that allows light from only a particular focal pointfrom the emission light being emitted by the sample to be detected bythe system, thereby producing high contrast, optically-sectioned imagesof relatively thicker samples than can be achieved by a SIM system. Theconfocal microscope is also capable of providing enhanced resolutionrelative to conventional wide-field microscopy. However, this enhancedimage resolution by the confocal microscope is attained by stopping downthe pinhole arrangement, which results in a corresponding prohibitiveloss in the fluorescence emission signal being detected from the sample.A modified confocal microscope has been shown to improve imageresolution to the resolution level of a SIM system without sacrificingemission signal strength; however, the slow scanning speed attained bythe microscope makes it impractical for research purposes.

As such, embodiments of the multi-focal SIM (MSIM) system as set forthherein include particular hardware components, properties andcharacteristics that address issues related to achieving high imageresolution at a high scanning speed and signal strength required byconventional microscopy imaging systems, and that provide betterperformance in thick samples than SIM systems that are currentlycommercially available. The MSIM system described herein includesvarious embodiments of hardware components that generate a multi-focalexcitation pattern for each image taken of the sample to produce a highresolution image at high scanning rates without significant signal lossrelative to conventional confocal microscopy and SIM systems. Inaddition, the MSIM system performs scaling, pinholing, and summing stepsusing just an arrangement of hardware components, such as pinholesmirrors, and micro-lens arrays rather than using a processor andsoftware arrangement to perform the same operation, as in structuredillumination microscopy. Further details of the multi-focal SIM systemsand methods are discussed in greater detail below.

Referring to the drawings, various embodiments of the multi-focal SIM(MSIM) system are illustrated and generally indicated as 100, 200, 300,400, and 500 in FIGS. 1-21. As illustrated in FIG. 1, one generalembodiment of the MSIM system, designated 100, is shown. MSIM system 100performs an illumination pattern generation operation 102, which splitsa single light beam into a plurality of light beams for generating oneor more multi-focal patterns, in which each multi-focal pattern definesan arrangement of focal points for each of the plurality of light beams.For example, a simplified illustration of one example of a multi-focalpattern 170 is shown in FIG. 5. In one embodiment, the multi-focalpattern 170 may include a plurality of focal points 172 arranged in aparticular multi-focal pattern 170 defined by the plurality of lightbeams with each multi-focal pattern 170 defining a different arrangementof focal points for the plurality of light beams such that multi-focalpattern 170 maximizes the distance between any two nearest focal points172 for a given density of focal points 172, thereby minimizingcrosstalk. As used herein, the term crosstalk refers to the ability ofnearby light beams to cause excitation and fluorescence from other focalpoints to appear as if focal points originate from the focal point inquestion. In a scanning operation 104, the plurality of light beams ineach multi-focal illumination pattern are rastered onto a sample 106being illuminated such that the sample 106 emits a plurality offluorescent emissions. The fluorescent emissions emitted by the sample106 are rastered in a de-scanning operation 108 which redirects theplurality of fluorescent emissions for removal of out-of-focusfluorescent emissions in a focusing operation 110. In the focusingoperation 110, out-of-focus fluorescent emissions are blocked and onlyin-focus fluorescent emissions are allowed to pass through forprocessing.

The in-focus fluorescent emissions caused by each multi-focal patternare then scaled using a scaling operation 112 that locally contractseach of fluorescent emissions by a predetermined factor. In oneembodiment of the scaling operation 112 illustrated in FIG. 6, a localcontraction of the fluorescent foci 172 caused in a single multi-focalpattern 170 occurs. For example, the local contraction of fluorescentfoci 172A and 172B to scaled fluorescent foci 172A′ and 172W,respectively, in the multi-focal pattern 170 is such that the distance700 between the geometric centers 175 of fluorescent foci 172A and 172Band the distance 702 of fluorescent foci 172A′ and 172B′ remains thesame regardless of the degree of scaling applied to the fluorescent foci172 of the multi-focal pattern 170. In other words, the scalingoperation 112 contracts the fluorescent foci 172 locally withoutinversion while keeping the relative distances between each of thefluorescent foci the same. After the scaling operation 112, the scaledin-focus fluorescent emissions for each multi-focal illumination patternare then rastered in a rescanning operation 114 that allows thecontracted, in-focus fluorescent emissions caused by each multi-focalpattern to be collected by a detector and summed to produce a compositehigh resolution image in a collection and summing operation 116.

In some embodiments, after the collection and summing operation 116 thecomposite image may undergo a deconvolution operation 118 that performsa level of de-blurring that further enhances the resolution of thecomposite image. The deconvolution operation 118 may be any conventionaldeconvolution operation 118, such as the freely available PiotyrWendykier's Parallel Iterative Deconvolution Plugin.

Referring to FIG. 2, another method for generating multi-focal patternsaccording to another embodiment of the MSIM system, designated 200, isillustrated. MSIM system 200 performs an illumination pattern generationoperation 202, in which a single light beam is split into a plurality oflight beams for generating one or more multi-focal patterns of theplurality of light beams. In a scanning operation 204, the plurality oflight beams for each multi-focal pattern is rastered onto a sample 206.The sample 206 produces fluorescent emissions in response to themulti-focal patterns that are detected in a collection operation 208. Inthe collection operation 208 a detector collects the in-focusfluorescent emissions and transmits the collected data to a processingsystem that performs a processing operation 210. The processingoperation 210 removes out-of-focus fluorescent emissions in eachmulti-focal pattern, scales the remaining in-focus fluorescentemissions, and then sums the scaled, in-focus fluorescent emissions togenerate a composite image composed from the plurality of fluorescentemissions created by the multi-focal patterns. In some embodiments, thecollected data may then undergo a deconvolution operation 212 similar tothe deconvolution operation 118, which performs a de-blurring of thecomposite image to further enhance image resolution.

Referring to FIG. 3, one embodiment of the multi-focal SIM system 100may include an illumination source 120, for example a laser, forgenerating a single light beam 122 that is transmitted through a beamsplitter 124, such as a micro-lens array. The beam splitter 124 splitsthe single light beam 122 into a plurality of light beams 126 with eachof the plurality of light beams 126 having a different focal point thatcollectively form a multi-focal pattern 128. In some embodiments, afirst lens 130 images each multi-focal pattern 128 through a dichroicmirror 132 in direction A and onto a scanning apparatus 134, such as agalvanometer, to perform the scanning operation 104.

During the scanning operation 104, the scanning apparatus 134 rasterseach multi-focal pattern 128 of the plurality of light beams 126 onto asample 144 through an arrangement of a second lens 136, a tube lens 138and an objective lens 140 and onto the sample 144.

In response to the sample 144 being illuminated by the multi-focalpatterns 128 of the plurality of light beams 126, the sample 144 emitsfluorescent emissions 142 caused by the multi-focal patterns 128composed of light beams 126. The plurality of fluorescent emissions 142for each multi-focal pattern 128 emitted by the illuminated sample 144is then captured through the objective lens 140, and passed through thetube lens 138 and second lens 136 and onto the scanning apparatus 134,which de-scans each of the plurality of fluorescent emissions 142 byrastering the fluorescent emissions 142 from the second lens 136 andonto the dichroic mirror 132 in a direction B opposite that of directionA in which the plurality of light beams 126 pass directly through thedichroic mirror 132. In direction B, the plurality of fluorescentemissions 142 are redirected by the dichroic mirror 132 to pass througha third lens 146 in which the plurality of fluorescent emissions 142 arethen focused onto a pinhole array 148 to perform the pinholingoperation.

During the pinholing operation, the pinhole array 148 physically blocksand rejects out-of-focus fluorescent emissions 142 and allows onlyin-focus fluorescent emissions 142A to pass through the pinhole array148. In one embodiment, the pinhole array 148 may include a plurality ofapertures configured to permit only in-focus fluorescent emissions 142Ato pass through the pinhole array 148, while blocking any fluorescentemissions 142 that do not pass through one of the apertures.

After passing through the pinhole array 148, the plurality of in-focusfluorescent emissions 142A are scaled using the scaling operation 112discussed above that locally contracts each of the focal points for arespective multi-focal pattern 128 by a predetermined factor, forexample a factor of two, using a first micro-lens array 150 arranged inseries with a second micro-lens array 152. In one embodiment, the firstmicro-lens array 150 collimates the multi-focal pattern 128 of in-focusfluorescent emissions 142A, while the second micro-lens array 152receives the collimated in-focus fluorescent emissions 142A and modifiesthe focal length for the collimated in-focus fluorescent emissions 142Asuch that each foci are scaled down by a predetermined factor to achievelocal contraction of the multi-focal pattern 128. In some embodiments,the scaled in-focus fluorescent emissions 142B caused by eachmulti-focal pattern 128 are then redirected by a first mirror 154 topass through a fourth lens 158 for focusing the scaled in-focusfluorescent emissions 142B onto a second mirror 156. The second mirror156 redirects the scaled in-focus fluorescent emissions 142B to passthrough an emission filter 160 that permits only scaled, in-focusfluorescent emissions 142B with a particular wavelength range, forexample 515 nm, to pass through the emission filter 160. Once thescaled, in-focus fluorescent emissions 142B are filtered, the scanningapparatus 134 rasters the fluorescent emissions 142B onto the detector164 through a fifth lens 162 such that the scaled, in-focus fluorescentemissions 142B for each multi-focal pattern 128 are collected and thein-focus fluorescent emissions 142B are summed by the detector 164 toproduce a composite image 143. The process of collecting the fluorescentemissions 142B caused by multi-focal patterns 128 is repeated until theentire field of view has been illuminated and all the resultingfluorescent emissions 142B are collected by the detector 164.

In some embodiments, the detector 164 may be a camera with the shutterleft open to exposure as the scaled, in-focus fluorescent emissions 142Bare rastered onto the detector 164 for collection until the entire fieldof view is illuminated and all the collected data is received.

In some embodiments, the composite image 143 is transmitted to aprocessing system 166 that performs the deconvolution operation 118 thatperforms a de-blurring function to enhance the resolution of eachcomposite image 143.

Referring to FIG. 4, one embodiment of the multi-focal SIM system 200for generating a multi-focal pattern 223 may include an illuminationsource 201 that generates a single light beam 203 that is transmittedthrough a beam expander 205 that produces an expanded single light beam203A, which is reflected off a beam steering mirror 207 for performingthe scanning operation 204 that scans the expanded light beam 203A ontoa beam splitter 209, for example a commercially available digitalmicromirror device (DMD) or a swept field confocal unit. In someembodiments, the DMD generates and switches multi-focal patterns 223with each focal point being an illuminated spot on the multi-focalpattern 223. Each illumination spot is created by a single DMD mirrorpixel being in the ON position such that a portion of the expanded lightbeam 203A is reflected off the single DMD mirror pixel. The beamsplitter 209 performs the illumination pattern generation operation 202that splits the expanded light beam 203A being scanned into a pluralityof expanded light beams 203B that collectively form a sequence ofmulti-focal patterns 223. The plurality of expanded light beams 203B foreach multi-focal pattern 223 is passed through a first tube lens 211 topass directly through a dichroic mirror 213 along a direction C.

After passing directly through the dichroic mirror 213 the expandedlight beams 203A are focused by an objective lens 215 onto a sampleplane for performing sample illumination 206 of sample 217 and generatea plurality of fluorescence emissions 222 emitted by the sample 217. Inthe collection operation 208, the fluorescent emissions 222 are focusedback through the objective lens 215 and onto the dichroic mirror 213along direction D perpendicular to that of direction C such that thefluorescent emissions 222 for each multi-focal pattern 223 areredirected and pass through a second tube lens 219. The second tube lens219 focuses the fluorescent emissions 222 onto a detector 221, thuscollecting each multi-focal pattern 223 in the form of collected data227 for transmission to a processing system 225 that rejectsout-of-focus light and performs scaling and summing operation 210. Theprocess of collecting the fluorescent emissions 222 caused by eachmulti-focal pattern is repeated until the entire field of view has beenilluminated. In some embodiments, the processing system 225 may includea processor 230 in operative communication with the detector 221 forprocessing collected data 227 stored in a database 232, such as acomputer-readable medium, by executing instructions a shall be discussedin greater detail below.

In one embodiment, the processing system 225 rejects out-of-focusfluorescent emissions 222, and performs the scaling and summingoperation 210 on the collected data 227 for each multi-focal pattern 223using a computerized procedure for processing the collected data 227.Referring to FIG. 7, a flow chart illustrates one method for performingthe out-of-focus fluorescence rejection, scaling and summing operation210 using the processing system 225. At block 1000, the processingsystem 225 performs an automatic lattice detection of the multi-focalillumination pattern 223. Specifically, automatic lattice detectiondetermines the position of each illumination focal point for aparticular multi-focal pattern 223 in order to perform pinhole maskingof each multi-focal illumination pattern 223 to produce masked images atblock 1002 and local contraction of the various fluorescent foci foreach masked image at block 1004. For example, the automatic latticedetection determines five vectors to completely specify the location ofall focal points for each fluorescent emission 222 in a particularmulti-focal pattern 223 in an image acquisition series. In someembodiments, the five vectors may be lattice vectors, shift vectors andan offset vector. Two lattice vectors specify the two-dimensionaldisplacement between any two neighboring illumination spots (e.g., focalpoints). As such, any lattice point displaced by a lattice vector willfall on another lattice point. Due to thermal shift, the exact locationsof each illumination spot may vary in time, so accurately extracting allvectors for each measured dataset from the collected data 227 providesthe best results using the following methodology. Finally, at block1006, summing each masked image to produce a composite image.

Referring to FIG. 8, a method for accurately extracting all vectors foreach measured dataset of collected data 227 is shown when the processingsystem 225 performs the out-of-focus rejection, scaling and summingoperation 210. The offset vector specifies the absolute position of theillumination spot closest to the center of a multi-focal pattern in anyone image acquisition series being processed by the processing system225. The shift vectors specify the distance of each illumination spotfor a particular multi-focal pattern 223 moves between consecutiveimages. Since the multi-focal pattern 223 is rastered in atwo-dimensional plane, a “fast” shift vector is applied at every step ofthe operation 210 and a “slow” shift vector is applied when a “fast”shift vector has completed illuminating one row. As used herein, theterm “fast” shift vector specifies those lattice points in a single rowand the term “slow” shift vector specifies successive rows. At block2000, each raw image representing the fluorescence 142B captured from aparticular multi-focal pattern 223 is multiplied by a Hann window (whichprevents ringing in the Fourier domain) and Fourier transformed. Atblock 2002, the resulting Fourier magnitudes are averaged to obtain ahigh signal-to-noise measurement of peak locations in the Fourier space.At block 2004, the averaged resulting Fourier magnitudes are filteredthrough a bandpass filter. At block 2006, a search for spikes or peaksin the Fourier domain is accomplished. Since the raw images for summedFourier magnitudes are expected to be a periodic lattice of peaks, thedimensions of each periodic lattice is made by determining the localFourier maxima, which are either lattice elements or noise. At block2008, the harmonics for the three lowest spatial frequency peaks aredetermined by computationally searching for peak intensities at and nearthe expected frequency multiples. At block 2010, given the locations ofthe three lowest spatial frequency peaks with harmonics, a search isconducted for peaks at their vector sums and differences, which composethe lattice vectors. In the present method, the peaks in a periodiclattice predict the location of other peaks in the same lattice withevery lattice point being located at the vector sum of an integer numberof Fourier lattice vectors. At block 2012, the processing system 225solves the resulting system of equations by linear squares to obtainthree lattice vectors. Any single peak in the Fourier domain will havesome positional error due to noise or pixelization. Utilizing the methodof least squares, the information provided by each peak produces a moreaccurate measurement of the lattice dimensions. At block 2014, theprocessing system 225 sums the lattice vectors to determine an errorvector and then subtracts one-third of the error vector from eachFourier lattice vector. At block 2016, any two of the two Fourierlattice vectors are selected and transformed to obtain real spacelattice vectors. It has been discovered that obtaining two latticevectors for the multi-focal illumination pattern makes it substantiallyeasier to measure the offset vector and the shift vector.

Referring to FIG. 9, a method for determining offset vectors is shownwhen the processing system 225 performs the rejection of out-of-focusblur, scaling and summing operation 210. At block 3000, a first rawimage is median filtered. At block 3002, a lattice of points from themeasured lattice vectors is constructed with an offset vector of zero.At block 3004, a set of smaller images centered at each lattice point isextracted from the median-filtered raw image using interpolation forsub-pixel centering. At block 3006, the smaller images are averaged toform a “lattice average” image. At block 3008, the brightest pixel inthe lattice average image is determined. At block 3010, the sub-pixelposition of each intensity peak in the lattice average image isestimated using interpolation techniques.

Referring to FIG. 10, a method for determining the shift vectors isshown when the processing system 225 performs the rejection ofout-of-focus blur, scaling and summing operation 210. At block 4000, aset of expected Fourier peak locations is constructed from the measuredFourier lattice vectors. At block 4002, extract phase is extracted foreach image number for the transformed collected data 227. At block 4004,the processing system 225 phase-unwraps each series of multi-focalpatterns 223 to remove 2rr jumps. It has been noted that the magnitudeof the Fourier transform of each raw image is independent of shifts ofthe raw images. Information about shifting is encoded in the phase ofthe Fourier transforms. At block 4006, the processing system 225reshapes each phase series in two-dimensional series to atwo-dimensional array with the same dimensions as the scan pattern(e.g., 16 pixels×14 pixels). After reshaping of each phase series, atblock 4008, a line is fit to the average slope of each phase series inboth the “fast” and “slow” directions. In each consecutive raw imagethat is detected, the illumination shifts by XFAsT. At the end of eachrow, the illumination also shifts by XsLow. If a raw image shifts by X,the phase of a peak in the Fourier space located at k shifts by k*X. Atblock 4010, the processing system 225 solves the resulting system ofequations for XFAST and XSLOW. At block 4012, an offset vector isconstructed for the first and last frames of the series of multi-focalpatterns 223. Computing the offset vector for the last frame, which usesthe same process that computed the offset vector of the first frame,provides a strong constraint that greatly increases the accuracy of theestimate. At block 4014, a predicted offset vector is constructed basedon the current estimate for XFAST and XSLOW. Once the predicted offsetvector is constructed, the difference between the two vectors isdetermined to be an error vector, which is divided by the number of rawimages in the scan, and subtracted from XsLow.

Referring to FIG. 11, a method for pinhole masking is shown forrejecting out-of-focus blur when the processing system 225 performs theout-of-focus blur, scaling and summing operation 210. At block 5000, aset of expected illumination locations based on the vectors determinedabove is constructed for each raw image collected during the collectionoperation 208. At block 5002, smaller subimages centered at eachillumination location are extracted using interpolation techniques andthen at block 5004 each centered subimage is multiplied by theprocessing system 225 by a two-dimensional Gaussian mask. These processsteps simulate the effect of physical pinholes. At block 5006, thebackground may be optionally subtracted from each subimage and thecentered subimage multiplied by a correction factor given by calibrationdata. At block 5008, individual hot pixels may be optionally medianfiltered by the processing system 225.

Referring to FIG. 12, a method for scaling and performing localcontraction is shown when the processing system 225 performs therejection of out-of-focus blur, scaling and summing operation 210. Atblock 6000, each masked subimage is re-sampled to shrink the maskedsubimage by a factor of about two. At block 6002, each scaled, maskedsubimage is added to the processing raw image.

Referring to FIG. 13, a method for summing is shown when the processingsystem 225 performs the rejection of out-of-focus blur, scaling andsumming operation 210. At block 7000, the processed raw images aresummed to form a composite image. At block 7002, the raw images may beoptionally summed to produce a calculated “widefield” image.

The resulting composite image produced by the rejection of out-of-focusblur, scaling and summing operation 210 has been shown to have a ⁻\12better resolution than a typical widefield image produced by aconventional widefield microscopy. The rejection of out-of-focus blur,scaling and summing operation 210 also greatly improves opticalsectioning, similar to a conventional spinning-disk or swept-fieldconfocal microscope.

Referring to FIG. 18, an embodiment of the MSIM system, designated 300,is shown. MSIM system 300 may be a swept-field hardware arrangement inwhich a similar degree of resolution enhancement as discussed above forMSIM systems 100 and 200 is achieved. The major difference between MSIMsystem 300 and MSIM systems 100 and 200 is that the operations relatedto scaling, pinholing, and summing steps are achieved in MSIM system 300using hardware rather than software as required for MSIM systems 100 and200. In this embodiment, MSIM system 300 may include a light source 302that emits a light beam 305 which is transmitted through a converging(+) microlens array 304 and the resulting focused light beam relayedthrough a first scan lens 306, scanned by a galvanometer mirror 308, andthen through a second scan lens 310 in which the first and second scanlenses 306 and 310 are in a 4f configuration to an intermediate imageplane.

As further shown, the galvanometric mirror 308 may be positioned at thefocal point between the first and second scan lenses 306 and 310 andsweeps the excitation foci across the sample plane 316, thus producing aswept-field excitation that covers the imaging field. In addition, atelescope arrangement of a tube lens 312 and an objective lens 314 ispositioned between second scan lens 310 and the sample plane 316 whichdemagnifies the intermediate stage and produces an array of excitationfoci which is swept across the sample plane 316 by the galvanometricmirror 308. The resulting fluorescence generated by the excitation focibeing swept across the sample plane 316 follows the same pathway backthrough the objective lens 314 and tube lens 312, which is descanned bythe galvanometer mirror 308, but diverted with a dichroic mirror 318positioned between the converging microlens array 304 and the first scanlens 306. Once diverted, the fluorescence emission is passed through apinhole array 320, thereby greatly reducing out-of-focus fluorescenceemission.

The resulting in-focus fluorescence emission is then relayed using a 4ftelescope pair consisting of a first relay lens 322 which focuses thein-focus fluorescence emission onto a first mirror 324 that diverts thein-focus fluorescence emission through a second relay lens 330 which isthen diverted by a second mirror 326 and a third mirror 328 insuccession to a second converging (+) microlens array 332. In oneembodiment, the second converging (+) microlens array 332 may bepositioned one focal length before the focus that would have been formedby the second relay lens 330, thereby producing an erect (non-inverted)image of the fluorescence emission foci with one half the magnification.This erect image is then relayed through another telescope arrangementof a third scan lens 334 and fourth scan lens 336 arranged in a 4 fconfiguration in which the erect image may be rescanned by thegalvanometer mirror 308 positioned at the focal point between the thirdand fourth scan lens 334 and 336. A camera 338 having an emission filter340 captures the final image. The MSIM system 300 can produce enhancedresolution images such as shown in FIG. 19 which shows AlexaFluor 488 nmlabeled microtubules in a fixed U2OS cell. The apparent width of each ofthe microtubules is about 200 nm. The image shown in FIG. 19 is rawwithout any post-processing deconvolution that would be expected tofurther increase the resolution of the image.

FIG. 20 shows an example of a spinning disk confocal microscopy system,designated 400. System 400 may include a light source 402 for generatinglight beams that are passed through a spinning disk with converging (+)microlens array 404 and then a spinning disk with a matched pinholearray 406. The resulting excitation is then imaged onto the sample 412with an optical arrangement of a tube lens 408 and objective lens 410.In this arrangement, the spinning of both the spinning disk withconverging (+) micolens array 404 and the spinning disk with matchedpinhole array 406 is performed in tandem, so that excitation foci thatare created cover the field of view of the sample 412. Fluorescenceemissions originating from the sample 412 are then passed back throughthe objective lens 410, the tube lens 408 and the spinning disk withmatched pinhole array 406. The fluorescence emissions are then divertedwith a dichroic mirror 414 such that the fluorescence emissions passthrough a telescope arrangement 416 which are captured by a camera 420through an emission filter 418.

Referring to FIG. 21, another embodiment of the MSIM system with aspinning disk multi-focal structured illumination microscopy which basedon the modified spinning disk system 400 described in FIG. 20,designated 500, is shown. In one embodiment, a conventional spinningdisk confocal microscope is converted into a resolution doubling devicewith one major modification. MSIM system 500 may include a light source502 that generates a light beam that is diverted by a dichroic mirror504 which passes through a spinning disk with converging (+) microlensarray 506 and then a spinning disk with a pinhole array 508 before beingfocused onto the sample 514 by a tube lens 510 and objective lens 512.Once the sample 514 is illuminated, the fluorescence emissions generatedby the sample 514 pass back through the objective lens 512 and tube lens510 which focus the fluorescence emissions through a spinning disk withdiverging (−) microlens array 516 positioned along the optic axis whichis made to spin in sync with the spinning disk with converging (+)microlens array 506 and the spinning disk with pinhole array 508 beforethe fluorescence emissions pass through a telescope arrangement 518before being captured by a camera 520 after being filtered by emissionfilter 522. In one embodiment, the spinning disk with diverging (−)micolens array 516 should contain the same number of micolenses as thespinning disk with converging (+) microlens array 506 and that thesemicrolenses should be spaced at the same spatial location on eachspinning disk 506 and 516. In some embodiments, if the microlenses forthe spinning disk with diverging (−) microlens array 516 have a halffocal length of the spinning disk with converging (+) microlens array506, and the spinning disks 506 and 516 are spaced apart by thedifference of the focal lengths, this arrangement will form a Galileantelescope with a magnification of ½. If such a telescope is positionedat the appropriate distance from the erect, demagnfied images are socreated, the rotation of the spinning disks 516 and 506 in synchronywith the camera exposure performs the desired pinholing, scaling, andsumming operations required for resolution-doubling. One importantadvantage of the spinning disk arrangement of the MSIM system 500 is thesignificantly reduced number of emission optics, nine instead ofseventeen. Another advantage is that if the spinning disk with diverging(−) microlenses array 516 can be readily fabricated, the optical set upfor the MSIM system 300, 400 and 500 is likely much easier to align thana conventional swept-field microscope, making the translation of theMSIM technology to spinning disk readily available.

Testing

To investigate the potential of the multi-focal SIM system 200 forbiological imaging, antibody-labeled microtubues in human osteosarcoma(U2OS) cells embedded in fluoromount were imaged. The following is adescription of the illumination system (FIG. 14), microscope system,sample preparation, and data processing of captured images. The use ofmulti-focal patterns in combination with deconvolution allowed us toinvestigate a variety of samples at imaging rates of 1 Hz, atresolutions down to 145 nm laterally and 400 nm axially. Compared toconventional structured illumination microscopy systems, the multi-focalSIM system 200 provided three-dimensional images of samples 5-8 timesthicker than conventional structured illumination microscopy systems. Inthe present investigation, microtubules were imaged in live transgeniczebrafish embryos at depths greater than 45 μm from the coverslipsurface. In addition, four-dimensional SIM datasets of GFP-labeledhistones in live nematode embryos were obtained.

For testing, a periodic lattice of approximately equilateral trianglesfor our illumination point locations because this particular patternmaximized the distance between any two nearest neighbors for a givendensity of points, thereby minimizing crosstalk. The multi-focalillumination pattern was translated one Digital Micromirror Device (DMD)pixel at a time, which corresponded to a step size of 120 nm in thesample plane. Larger steps did not evenly illuminate the sample, givinga visible striping artifact, while smaller steps increased acquisitiontime and dose with no increase in image quality.

Multi-focal patterns were imaged onto the sample, which was mounted on acommercial inverted microscope, and a scientific-grade complementarymetal-oxide-semiconductor camera (sCMOS) was used to record one rawimage for each multi-focal pattern position. By varying the spacingbetween the illumination points, acquisition speed may be traded forsectioning quality. It was discovered that widely spaced foci had lesscrosstalk, but additional multi-focal illumination patterns wererequired to evenly illuminate a sample. In contrast, denser foci hadmore crosstalk, but required correspondingly fewer multi-focal patternsto evenly illuminate the sample. It was found that a multi-focal patternwith a 16 pixel horizontal and a 14 pixel vertical separation betweenscan points provided good results in the biological samplesinvestigated. The resulting 224 raw exposures taken at 222 Hz for a 480pixel×480 pixel field of view corresponded to about a 1 Hzsuper-resolution image acquisition rate.

To investigate the potential of multi-focal SIM system 200 forbiological imaging, we imaged antibody-labeled microtubules in humanosteosarcoma (U2OS) cells embedded in fluoromount as shown in the imagesillustrated in FIG. 15. Compared to widefield images, the multi-focalilluminated, pinholed, summed, and scaled images produced by the SIMsystem 200 improved image resolution including image contrast. Inaddition, parallel, iterative deconvolution further improved theresulting composite image, which revealed features previously obscuredby diffraction. The apparent full-width at half maximum (FWHM) intensityof microtubules in multi-focal SIM system 200 images was 145 nm, whichwas a two-fold improvement compared to widefield imaging. Similarexperiments on 110 nm subdiffractive beads confirmed this result(multi-focal SIM system 200 FWHM 146+/−15 nm vs. widefield FWHM 284+/−32nm, N=80 beads. The total acquisition time for the 48 μm×49 μm field wasabout 1 s, a 6500-fold improvement over a conventional image scanningmicroscopy (ISM) system assuming the same 222 Hz raw frame rate for eachmicroscopy system.

The suitability of the multi-focal SIM system 200 for dual-labeled,three-dimensional samples was also investigated as shown in the samplesillustrated in FIG. 16. A Z stack of images on a fixed cell embedded influoromount was used. Immuno-labeled microtubules with Alexa Fluor 488and stained mitochondria with Mitotracker Red obtained a volume of 3.7μm thickness, with individual slices separated by 100 nm. Compared towidefield images obtained with the same total illumination dose,three-dimensional multi-focal SIM system 200 images provided a strikingincrease in image contrast, due to the combined physical (via digitalpinholes) and computational (via three-dimensional deconvolution)removal of out-of-focus light.

The resulting composite images produced by the multi-focal SIM system200 had approximately a two-fold resolution improvement over widefieldimaging; better resolving microtubules and “worm-like” mitochondria. Forexample, better resolving of sub-diffractive voids at the ends ofindividual mitochondria was achieved including microtubule pairsseparated by greater than 200 nm. Unexpectedly, multi-focal SIM system200 also improved the axial resolution approximately two-times overwidefield images, as microtutubles had apparent axial FWHM of about 400nm. This result was confirmed on 100 nm subdiffractive beads(Multi-focal SIM system FWHM 402+/−49 nm; widefield 826+/−83 nm, N=80beads.

The MSIM system 200 was also applied to three-dimensional imaging ofthicker live samples in which the pinhole operation physically rejectsout-of-focus light that would otherwise swamp the in-focus light signal.To demonstrate this capability, live, immobilized zebrafish embryosexpressing a GFP transgene that labeled microtubules were imaged.

Using multi-focal illumination in accordance with the multi-focal SIMsystem 200, 241 slices were acquired spaced 0.2 μm apart at atwo-dimensional imaging rate of 1 Hz. After pinhole focusing, scaling,and three-dimensional deconvolution, a volume of 48.2 μm thickness wasachieved as shown in the images of the sample illustrated in FIG. 17.Structural features such as the boundary between two adjacent somites,alignment of microtubules along the somite boundary, and microtubulefree-regions corresponding to the nuclei of the developing muscle cellsare clearly visible in the stack. Estimation of the z-position ofsuccessively deeper nuclei within this stack suggests that the imagingvolume contained 6-7 cell layers.

In one test, the imaging rate of the multi-focal SIM system 200 captureda dividing cell in the epidermis without significant motion blur in theimages. The resolution enhancement of multi-focal SIM system 200 wasretained throughout the volume, as the separation between microtubulepairs at the site of the cell division was resolved to better than 200nm, and microtubules in the epidermis had lateral FWHM 175+/−33 nmlaterally (N=30) and 496+/−65 nm axially (N=21).

Illumination System

In the illumination system, all optics were mounted on an optical table(Kinetic Systems, Vibraplane Model #5704-3660-23SPL) to minimizemechanical vibrations. For exciting fluorescence, two lasers were used:a 150 mW, 561 nm laser (561, Coherent, Sapphire 561-150 CW CDRH) and a200 mW, 488 nm laser (488, Coherent, Sapphire 488-200 CDRH). Mechanicalshutters (Thorlabs, SHO5 and SC10) placed after each laser was used tocontrol illumination. Beams were combined with a dichroic mirror (DC,Chroma, 525dcxru) and expanded 6.7 times with a beam expanderconstructed from two achromatic lenses (Edmund, f=30 mm, NT49-352-INKand Thorlabs, f=200 mm, AC254-200-A-MLO. Expanded beams were directedonto a digital micromirror device (DMD, Digital Light Innovations, D4100DLP 0.55″ XGA) 24 degrees off normal, so that in the ON position themicromirrors tilted the output beam normal to the DMD face. The centerorder of the resulting illumination pattern was demagnified 1.5 timeswith a beam de-expander (Thorlabs, f=75 mm, AC254-075-A-ML and f=50 mm,AC254-050-A-MLO, aligned in a 4 f configuration such that the DMD facewas re-imaged at the back focal plane of a 180 mm tube lens internal tothe microscope (Olympus, IX-81). These elements are shown in FIG. 14.After entering the left side port of the microscope, the beamsequentially passed through (i) the tube lens; (ii) a dichroic mirror(Chroma, zt405/488/561); (iii) a 60× objective (Olympus, PlanApo, NA1.45 TIRF, for single cells, or UPLSAPO 60XS, NA 1.3, for zebrafish andworm embryos) for a total demagnification of 90× between the DMD and thesample being illuminated. The illumination at the sample covered acircular region approximately 50 μm in diameter.

Microscope System

Structured illumination microscopy (SIM) imaging was performed on anOlympus IX81 inverted microscope equipped with both left and right sideports, and an automated XY stage with an additional Z piezoelectricstage (200 μm range, Applied Scientific Instrumentation, PZ-2000). Thepatterned excitation (e.g. multi-focal illumination pattern) created bythe DMD was brought in via the left side port to the microscope.Fluorescence emitted by the illuminated sample was collected by theobjective, reflected with a dichroic mirror (Chroma, zt405/488/561),passed through a 180 nm tube lens internal to the microscope, filteredappropriately to reject pump light (Semrock, LP02-488RE-25 and NF03-561E-25), and detected with a scientific-grade complementarymetal-oxide-semiconductor (sCMOS) camera (Cooke, pco.edge) mounted onthe right side port. Correctly aligning the sCMOS along the optical axiswas critical in achieving near diffraction-limited performance. To aidin the correct positioning of the camera, a 60× objectives typicallyused in imaging with a 10× air objective (Olympus, CPlanFl 10×, 0.3 NA),an optic much more sensitive to errors in axial alignment. A fixedillumination pattern (similar to one used in SIM) onto the fluorescentlake sample, and translated the camera along the optical axis until theapparent size of each illumination spot was minimized.

Sample Preparation

U2OS cells were cultured on ethanol sterilized, cleaned #1.5 25 mmdiameter coverslips (Warner Instruments, 64-0715) in standard growthmedia (DMEM-HG (Invitrogen, 11960), sodium pyruvate (Invitrogen, 11360),GlutaMAX (Invitrogen, 35050) and 10% heat inactivated fetal bovine serum(Invitrogen, 11082)). To stain the samples for microtubules, cells werefixed in with a mixture of 0.5% glutaraldeyde, 0.37% formaldehyde, and0.3% Triton X-100 in Cytosketetal Buffer (CB, 10 mM MOPS, 138 mM KCl, 2mM MgCl₂, 2 mM EGTA, 0.01% NaN₃, and 160 mM Sucrose, pH 6.1). Afterfixation, the cells were washed in CB, quenched with 100 nm glycine,washed in CB, and blicked in antibody dilution buffer (AbDil, 150 mMNaCl, 20 nM Tris, 0.1% Triton X-100, 0.1% NaN₃, and 2% bovine serumalbumin, pH 7.4). The primary monoclonal antibody (Invitrogen, 32-2500)was incubated with the cells diluted to 2 μg/mL in AbDil for one hour atroom temperature. Following primary antibody incubation, the cells werewashed in the phosphor-buffered saline before incubating the cells withthe secondary, Alexa Fluor 488 labeled antibody (Invitrogen, A-11001) at1:200 dilution in AbDil for 1 hour.

Samples for dual-color experiments were initially stained withMitotracker Red (Invitrogen, M-7512) as per the manufacturer'sinstruction prior to fixation. After mitochondrial labeling, theprocedure outline above was used to stain the microtubules. All sampleswere mounted in fluoromount G (Electron Microscopy Solutionis, 17984-25)to a standard 25 mm×75 mm glass slide (SPI supplieds, #01251-AB) andsealed with nail polish.

a) Subdiffractive Beads

Yellow-green or red fluorescent beads (Invitrogen, F8803, 110 nmdiameter; Invitrogen F8801, 100 nm diameter) were used for all pointspread function (PSF) measurements. Beads were diluted from the stockconcentration of 1:1300 (1:200 in distilled water and 1:13 in ethanol)and spread over cleaned glass coverslips. After air-drying for 5 minutesto evaporate the ethanol, coverslips were washed twice in distilledwater to remove unattached beads. After air-drying again, the beads weremounted in fluoromount or silicone oil onto glass slides, and sealedwith nail polish.

a) Zebra Fish Samples

Tg(XlEefl al: dclk2-GFP)^(io008) embryos carrying the zebrafishdclk2-GFP transgene were used in thick MSIM experiments shown in FIG.16. To construct, this line, plasmids containing the transgene wereinjected into one-cell zebrafish embryos along with Toll mRNA.Fluorescent embryos were raised to adulthood and crossed to select forgermline transmission by screening the offspring for GFP expression.

Tg(XlEdflal:dclk2-GFP)^(io008) embryos were collected by naturalspawning and maintained at 28 degrees Centigrade. Prior to imaging bythe multi-focal SIM system 200, embryos at 24 hpf were anesthetized inTricaine (Sigma, E105210 at a final concentration of 600 μM in embryomedia (60 mg Instant ocean sea salt (Petsmart) per liter ddH₂O).Anesthetized embryos were mounted on round coverslips, immobilized in 1%low-melt agarose (Cambrex, 50080), placed in a round coverslip holder(ASI, I-3033-25D), covered with embryo media, and imaged at roomtemperature.

Data Processing

Following acquisition of raw images using the illumination andmicroscope system described above, each set of collected raw images ofthe samples were processed into a super-resolution image using theprocessing system 225 having software written in the Python programminglanguage. The processing steps employed by the processing system 225were: (i) Automatic lattice detection to precisely determine therespective locations of the illumination spots; (ii) Digital pinholemasking around each detected illumination spot to reject out-of-focuslight, and optical flat-fielding using calibration data; (iii) Localcontraction (e.g., scaling), and re-sampling the area around eachillumination spot to improve the resolution by ⁻Ni2; (iv) Summing theprocessed raw images to produce a super-resolution composite image; and(v) Using conventional deconvolution techniques to recover the full 2×resolution enhancement. These process steps are discussed in greaterdetail above with respect to the focusing, scaling, and summingoperation 210 executed by the processing system 225 of the multi-focalSIM system 200.

It should be understood from the foregoing that, while particularembodiments have been illustrated and described, various modificationscan be made thereto without departing from the spirit and scope of theinvention as will be apparent to those skilled in the art. Such changesand modifications are within the scope and teachings of this inventionas defined in the claims appended hereto.

What is claimed is:
 1. A microscopy system (300) comprising: a lightsource (302) for transmitting a single light beam (305); a firstmicrolens array (304) for splitting the single light beam (305) into aplurality of light beams for forming a multi-focal pattern; a scanner(308) for scanning the plurality of light beams that forms themulti-focal pattern onto a sample such that the sample generates aplurality of fluorescent emissions with the multi-focal pattern; apinhole array (320) to block out-of-focus fluorescent emissions for themulti-focal pattern and allowing in-focus fluorescent emissions to passthrough the pinhole array (320); a first relay lens (322) and a secondrelay lens (330); a first mirror (324), a second mirror (326), and athird mirror (328); a second microlens array (332), wherein the firstrelay lens (322) focuses the in-focus fluorescent emissions onto thefirst mirror (324) which diverts the in-focus fluorescent emissionsthrough the second relay lens (330) and to the second microlens (332)via the second mirror (326) and the third mirror (328) to produce anon-inverted image of the in-focus fluorescent emissions having a onehalf magnification, wherein the scanner (308) rescans the non-invertedimage of the in-focus fluorescent emissions; and a camera (338) forcapturing the scanned non-inverted image; wherein a first scan lens(306) and a second scan lens (310) are in 4f configuration to anintermediate image plane.
 2. The microscopy system (300) of claim 1,wherein the first scan lens (306) and the second scan lens (310) areconfigured for focusing the plurality of light beams from the firstmicrolens array (304) to an intermediate image plane.
 3. The microscopysystem (300) of claim 2, further comprising: an objective lens (314) andtube lens arrangement (312) positioned between the scanner (308) and thesample for demagnifying the intermediate image plane of the plurality oflight beams and producing an array of excitation foci from the pluralityof light beams for each of the multi-focal patterns across the sample.4. The microscopy system (300) of claim 1, wherein the scanner (308)comprises a galvanometric mirror.
 5. The microscopy system (300) ofclaim 1, wherein the scanner (308) is positioned at a focal pointbetween the first scan lens (306) and the second scan lens (310),wherein the first scan lens (306) is positioned between the firstmicrolens array (304) and the scanner (308) while the second scan lens(310) is positioned between the scanner (308) and the sample.
 6. Themicroscopy system (300) of claim 1, wherein the first microlens array(304) and the second microlens array (332) are converging microlenses.7. The microscopy system (300) of claim 1, further comprising: a thirdscan lens (334) positioned between the second microlens array (332) andthe scanner (308) and a fourth scan lens (336) positioned between thescanner (308) and the camera (338).
 8. The microscopy system (300) ofclaim 1, wherein the non-inverted image is relayed through a third scanlens (334) and a fourth second scan lens (336) arranged in 4 fconfiguration to image the intermediate image plane.
 9. The microscopysystem (300) of claim 8, wherein the non-inverted image is rescanned bythe scanner (308) and captured by the camera (338).
 10. The microscopysystem (300) of claim 1, wherein the second microlens (332) ispositioned one focal length before the focus that would have been formedby the second relay lens (330) if the second microlens (332) was absentfrom microscopy system (300).